How do you Behave as a Psychometrician? Research Conduct in the Context of Psychometric Research

The identification of fraudulent and questionable research conduct is not something new. However, in the last 12 years the aim has been to identify specific problems and concrete solutions applicable to each area of knowledge. For example, previous work has focused on questionable and responsible research conducts associated with clinical assessment, measurement practices in psychology and related sciences; or applicable to specific areas of study, such as suicidology. One area of study that merits further study of questionable and responsible research behaviors is psychometrics. Focusing on psychometric research is important and necessary, as without adequate evidence of construct validity the overall validity of the research is at least debatable. Our interest here is to (a) identifying questionable research conduct specifically linked to psychometric studies; and (b) promoting greater awareness and widespread application of responsible research conduct in psychometrics research. We believe that the identification and recognition of these conducts is important and will help us to improve our daily work as psychometricians. (AU)

O futuro das avaliações por meio das redes sociais / The future of assessments through social media / El futuro de las evaluaciones mediante las redes sociales

O objetivo geral da presente pesquisa é discutir as principais inovações em avaliação psicológica possibilitadas pela existência das redes sociais. Mais especificamente, iremos caracterizar as práticas (e fragilidades) da avaliação psicológica convencionais, os métodos mais modernos de psicometria computacional e exemplos de aplicações de psicometria computacional a partir de dados provenientes de redes sociais. De forma geral, as práticas convencionais da avaliação psicológica da psicometria foram criadas no século XIX e muitas de suas práticas, desenvolvidas próximo à metade do século XX, continuam sendo utilizadas de forma pouco crítica. Como alternativa, a psicometria computacional, uma abordagem da psicometria que combina métodos da ciência da computação orientados a dados e teoria psicométrica, tem sido utilizada para gerar inovações na área de avaliação. Por fim, discutimos algumas aplicações da psicometria computacional e como essas inovações irão, provavelmente, gerar mudanças profundas no contexto de avaliação.(AU)

The overall aim of this study was to discuss the main innovations in psychological assessment made possible by the existence of social networks. More specifically, we characterize the practices (and weaknesses) of conventional psychological assessment, the most modern methods of computational psychometry and examples of applications of computational psychometry using data from social networks. In general, the conventional practices of psychological assessment of psychometrics were created in the 19th century and many of its practices, developed around the middle of the 20th century, continue to be used in a non-critical way. As an alternative, computational psychometrics, an approach to psychometrics that combines data-driven methods of computer science and psychometric theory, has been used to generate innovations in the area of assessment. Finally, we discuss some applications of computational psychometrics and how these innovations are likely to generate profound changes in the assessment context.(AU)

El objetivo general de esta investigación es discutir las principales innovaciones, en materia de evaluación psicológica, posibilitadas por la existencia de las redes sociales. Más específicamente, caracterizaremos las prácticas convencionales (y debilidades) de la evaluación psicológica, los métodos más modernos de psicometría computacional y ejemplos de aplicaciones de psicometría computacional a partir de los datos de redes sociales. En general, las prácticas convencionales de evaluación psicológica de la psicometría se crearon en el siglo XIX y muchas de sus prácticas, desarrolladas a mediados del siglo XX, continúan utilizándose de forma acrítica. Como alternativa, la psicometría computacional, un enfoque de la psicometría que combina métodos informáticos basados en datos y teoría psicométrica, se ha empleado para generar innovaciones en el área de la evaluación. Por último, discutimos algunas aplicaciones de la psicometría computacional y cómo es probable que estas innovaciones generen cambios profundos en el contexto de la evaluación.(AU)

Are we measuring the same thing? Psychometric and research considerations when adopting new testing modes in the time of COVID-19.

As the world navigates unchartered territories and witnesses the overwhelming impact of COVID-19, investigators face important challenges to ensure continuity of research studies in a scientifically sound manner. Given the susceptibility of the older population to COVID-19, research in the field of aging and dementia may be more severely impacted than other areas. With in-person testing halted, researchers are considering remote testing to collect data on questionnaires and functioning, including cognitive functioning. This is not without challenges. Here, we discuss psychometric properties of the scales that need to be considered and evaluated when implementing remote testing to ensure the quality of the studies is preserved. We encourage the community to join efforts to improve practice sharing and facilitating access to item-level data.

Is Bullying Associated with Emerging School Refusal in Autistic Boys?

The experience of being bullied is widespread among autistic youth. Relatively little empirical work has been done on the relationship between the bullying of these youth and school refusal (SR). This study of 67 school-age autistic boys (M = 11.7 years, SD = 2.3 years) examined several factors that may contribute to SR. Data regarding boys' age, generalised anxiety disorder (GAD), major depressive disorder (MDD), key ASD diagnostic criteria, and frequency of being bullied were collected. Results indicated that, while boys displaying emerging SR also had significantly higher GAD and MDD than boys without emerging SR, only the frequency of being bullied made a significant contribution to emerging SR. Implications for prevention and treatment of SR among autistic youth are discussed.

Quality of acute internal medicine: A patient-centered approach. Validation and usage of the Patient Reported Measure-acute care in the Netherlands.

BACKGROUND: Providing high quality care is important and has gained more attention since the introduction of value-based healthcare. Value should be measured by outcomes achieved, relevant for patients. Patient-centeredness is one domain for quality improvement determined by the Institute of Medicine, aiming to deliver care responsive to the patient. The development and implementation of patient reported outcome- and experience measures can be used for this goal. Recently, we developed the Patient Reported Measure (PRM)-acute care, based on five relevant domains to evaluate and improve the quality of care in the Emergency Department (ED). OBJECTIVE: To validate the PRM-acute care, in order to evaluate and improve patient-centered care in the ED. METHODS: We performed a prospective questionnaire-based study. Patients ≥18 years presenting for internal medicine in the ED were eligible. The validity of the PRM-acute care was evaluated according to the COSMIN-criteria. We performed hypotheses testing to evaluate construct validity. The perceived quality of care was evaluated by statistical analysis. RESULTS: Face- and content validity was evaluated based on previously performed research and deemed good. Construct validity was supported by demonstrated differences between subgroups; patients with severe symptoms had a higher perceived quality of care. The correlation between overall satisfaction and the total mean score of the PRM-acute care (r = 0,447, p = 0.01) was significant. Overall, patients reported a mean perceived quality of care of 4.67/6.0. CONCLUSION: The PRM-acute care is a valid instrument to measure the perceived quality of care in an acute setting for internal medicine patients. Additionally, patients reported a good perceived quality of care in the ED with scores ranging from moderate to well for each of the relevant domains. Therefore, we believe that the PRM-acute care can be implemented in daily practice to evaluate the perceived quality of care and to improve the quality of acute care.

Development and Psychometrics Test of Home Environment Assessment Checklist for Community-Dwelling Older Adults with Dementia.

BACKGROUND: Home environment is a core domain in the care of community-dwelling older adults with dementia, but there is no suitable instrument to measure it in China. OBJECTIVE: To develop and psychometrically test the home environment assessment checklist for community-dwelling older adults with dementia. METHODS: A three-step process was performed to develop and test this instrument: 1) based on the evidence-based theory, the checklist was summarized as the main points of evidence from living environment settings among older adults with dementia, 2) the draft tool was assigned to an iterative process of evaluation by a panel of examiners consisting of experts from treatment, nursing and caring, people with dementia and their caregivers, 3) inter-rater reliability and internal consistency were calculated with a sample of 348 caregivers of the older adults with dementia. RESULTS: The HEAC consisted of 71 items in domains addressing safety, stability and familiarity, visual cues, and sensory stimulation. Psychometric evaluation showed that this tool demonstrated sound reliability and validity. Content validity was 0.969 which was established by a panel of experts (n = 10). Inter-rater reliability of two researchers was 0.978, and 0.848 for researchers and caregivers. Test-retest reliability was excellent (ICC = 0.757-0.877) in community-dwelling older adults with dementia 2 week apart. CONCLUSION: The HEAC is a new tool to help collect the reliable information on the barriers and facilitators of home environment for community-dwelling older adults with dementia and to precipitate the home modification process to improve the quality of care for people with dementia and their caregivers in daily life.

Development and validation of the incontinence associated dermatitis knowledge, attitude and practice questionnaire.

AIM: This study aimed to develop and test the validity and reliability of the Knowledge, Attitudes and Practices of Incontinence-associated Dermatitis Questionnaire (KAP-IAD-Q) for Nurses. METHODS: A psychometric validation design was employed. Phase I of the study entailed the development of items through an extensive literature review and a double Delphi procedure with 11 experts specialised in wound, ostomy and continence to examine content validity of the KAP-IAD-Q. Phase II involved administering the KAP-IAD-Q to a convenience sample of 263 Registered Nurses from a public hospital in Singapore to evaluate its construct validity, internal consistency and test-retest reliability. RESULTS: The instrument showed acceptable content validity (S-CVI = 0.85). Exploratory factor analysis showed all 22 items demonstrated strong factor loadings >0.4 and the four factors KAP-IAD-Q explained 58.1% of total variance. The four factors were☹1) knowledge om IAD aetiology and identification, (2) knowledge on IAD risk factors; (3) attitudes, and (4) practices. The overall internal consistency was excellent (Cronbach's α = 0.913). The KAP-IAD-Q showed good overall test-retest reliability as well (ICC = 0.89 (95% CI 0.69-0.96, p < 0.001). CONCLUSION: The KAP-IAD-Q demonstrated good psychometric properties and is effective in measuring levels of IAD-related KAP among nurses. Further confirmation of the proposed factor structure is recommended. Future research should explore determinants of nurses' KAP and associations between IAD knowledge, attitudes and practices.

Development and validation of the Standing Balance Assessment for Individuals with Spinal Cord Injury (SBASCI) - A new outcome measure.

BACKGROUND: Injury to the spinal cord results in standing balance impairment following variable sensorimotor loss. Standing balance training is a realistic goal for the majority of individuals with spinal cord injury (SCI) for which therapists need valid measures to assess standing ability in people with SCI that are relevant to functionality. OBJECTIVE: The objective of the study was to develop an all inclusive Standing Balance Assessment for Individuals with Spinal Cord Injury (SBASCI) measure and to establish its initial psychometric properties. METHODS: The study was carried out in three phases: Item development, scale development and scale evaluation. Literature review, focus group discussions and evaluation by experts resulted in the development of a 22-item SBASCI scale. The scale was administered on 120 participants with SCI. Exploratory factor analysis and item analysis were used to determine construct validity and internal consistency of the scale. RESULTS: Content validity was established qualitatively and quantitatively. The scale shows high internal consistency reliability (Cronbach's alpha 0.96). The results of the exploratory factor analysis suggested a four factor structure retaining all the 22 items. CONCLUSION: SBASCI is a valid and reliable scale to measure the standing balance of individuals with SCI. Further studies are required to establish other psychometric properties.

Development of the short version of the spielberger state-trait anxiety inventory.

The Spielberger State-Trait Anxiety Inventory (STAI) has been widely used to measure the state and trait components of anxiety. We sought to develop a short, yet reliable and valid form of these scales for use in circumstances where the full-form is not feasible. We abbreviated the scales using item response theory analyses to retain the items that could discriminate the best among participants. One sample (N = 922) completed the state scale, a second sample (N = 2227) completed the trait scale, while a third sample (N = 250) completed the short forms. Our participants completed the Hungarian version of STAI alongside other measures to observe external validity. We calculated cut-off scores for the state (>9.5,) and trait (>13.5) scales. A total of 19.5% and 20.1% of the respondents reached the cut-off scores. The five-item short forms of STAI had sound psychometric properties that are comparable to those obtained on the full-form. The external validity of the scales is also demonstrated. We report detailed descriptive statistics that could be used in further studies as standards. The short scales are reliable measures that could be used in clinical screening and behavioural research; especially where practical considerations preclude the use of a longer questionnaire.

Cognition-tracking-based strategies for diagnosis and treatment of minimal hepatic encephalopathy.

Minimal hepatic encephalopathy (MHE), which shows mild cognitive impairment, is a subtle complication of cirrhosis that has been shown to affect daily functioning and quality of life. However, until 2014, relevant guidelines do not give much attention to the diagnosis and treatment of MHE, resulting in patients being ignored and denied the benefits of treatment. In this review, we summarize recent cognition-based research about (1) alteration of nerve cells, including astrocytes, microglial cells and neurons, in mild cognitive impairment in MHE; (2) comparison of methods in detecting cognitive impairment in MHE; and (3) comparison of methods for therapy of cognitive impairment in MHE. We hope to provide information about diagnosis and treatment of cognitive impairment in patients with MHE.

Informatics and Cognitive Assessment: A RUDAS Scale Paradigm.

BACKGROUND: Cognitive assessment is an essential element of the screening process of Alzheimer's disease. The prevalence of dementia is increasing and so are the numbers of immigrants and elderly population relocating and in need for health diagnosis and treatment. However, most of the psychometric tools used in psychological assessments are time-consuming and suffer from biases of language and cultural restrictions. OBJECTIVES: Our objective was to create a computerized version of a multicultural cognitive screening test, which would simplify cognitive assessment of elderly multicultural population, as routine part of health check-up procedures. METHODS: The application was implemented in Android Studio and was written in Java code with the use of a home PC and a tablet. Rowland Universal Dementia Assessment Scale (RUDAS) was chosen. RUDAS is a cognitive screening tool with good psychological characteristics, which was created for multicultural and bilingual populations and was free to download. The collaboration with an authorized psychologist was essential for the ethics of the psychometric science. RESULTS: The complete computerized version of RUDAS will include the six-item questionnaire, assessing specific cognitive domains which are in high correlation with Alzheimer's screening process, such as registration, visuospatial orientation, praxis, visuo-constructional drawing, judgment, memory recall and language. CONCLUSION: The utilization of informatics in making cognitive assessment a user-friendly, validated, not time- or cost-consuming procedure would add value to psychometric tools which still are administered with "pen and paper", when this proceeds with respect to the ethics of the science.

Conceptualization and Measurement of Adolescent Prosocial Behavior: Looking Back and Moving Forward.

The struggle to cast a net around the numerous ways prosocial behavior is expressed lends itself to the absence of widely accepted methods of measurement. Additionally, research intent on evaluating the psychometric properties of current approaches has been somewhat limited. Weaving together seminal as well as contemporary research, the current review focuses on how these conceptual and measurement issues pertain to adolescent studies (in an intentional effort to offset the somewhat disproportionate focus directed toward prosocial development in infants, children, and adults). Recommendations to address current limitations and attain a more nuanced understanding of the construct are presented and discussed.

New insights on technology and assessment: introduction to JWOP special issue

No disponible

Cancer screening risk literacy of physicians in training: An experimental study.

We investigated what factors may foster or hinder physicians' cancer screening risk literacy-specifically the ability to understand evidence regarding screening effectiveness and make evidence-based recommendations to patients. In an experiment, physicians in training (interns and residents) read statistical information about outcomes from screening for cancer, and had to decide whether to recommend it to a patient. We manipulated the effectiveness of the screening (effective vs. ineffective at reducing mortality) and the demand of the patient to get screened (demand vs. no demand). We assessed participants' comprehension of the presented evidence and recommendation to the patient, as well as a-priori screening beliefs (e.g., that screening is always a good choice), numeracy, science literacy, knowledge of screening statistics, statistical education, and demographics. Stronger positive a-priori screening beliefs, lower knowledge of screening statistics, and lower numeracy were related to worse comprehension of the evidence. Physicians recommended against the ineffective screening but only if they showed good comprehension of the evidence. Physicians' recommendations were further based on the perceived benefits from screening but not on perceived harms, nor the patient's demands. The current study demonstrates that comprehension of cancer screening statistics and the ability to infer the potential benefits for patients are essential for evidence-based recommendations. However, strong beliefs in favor of screening fostered by promotion campaigns may influence how physicians evaluate evidence about specific screenings. Fostering physician numeracy skills could help counteract such biases and provide evidence-based recommendations to patients.

The psychometric properties of a new oral health illness perception measure for adults aged 62 years and older.

BACKGROUND: Based on the Common-Sense Model of Self-Regulation (CSM), a new integrated Illness Perception Questionnaire Revised for Dental Use in Older/Elder Adults (IPQ-RDE) was developed for single and multiple dental conditions. This study describes psychometric properties of the IPQ-RDE for adults 62 years and older. METHODS: Participants (n = 198) living in 16 subsidized housing facilities completed the IPQ-RDE and a questionnaire assessing their socio-demographics, frequency of dental visits, perceived condition of teeth/gums, depression, social support, and oral health quality of life (OHQOL). Participants received dental screening for presence/absence of teeth, coronal and root caries, and periodontitis. The 43-item IPQ-RDE was tested for internal (construct, discriminant) and external validity (concurrent, construct, discriminant, predictive) and reliability (internal consistency). RESULTS: Confirmatory factor analysis demonstrated that a ten-factor model in accordance with the CSM framework (identity, consequences, control, timeline, illness coherence, treatment burden, prioritization, causal relationship, activity restriction, emotional representations) had good construct validity based on significant factor loadings and acceptable model fit (RMSEA = 0.065, CFI = 0.902). Edentulous participants had significantly higher mean factor scores (inaccurate perception) for overall IPQ-RDE and four constructs indicating concurrent validity. Discriminant validity was suggested by non-relationship with external measures (education, dental visit frequency). Predictive validity was indicated by the negative correlation of most constructs with OHQOL suggesting that inaccurate perception was related to lower quality of life. Internal consistency of eight IPQ-RDE constructs was excellent (Cronbach's alpha > 0.73). CONCLUSIONS: The IPQ-RDE is a valid and reliable new measure for assessing older adult's perception of dental conditions. It can be an important tool for oral health behavioral research to restructure older adult's perception of dental conditions, and subsequently prevent tooth loss and improve oral health quality of life.

Preliminary psychometric analyses of the International Spinal Cord Injury Quality of Life Basic Data Set.

STUDY DESIGN: Secondary psychometric analysis of cross-sectional previously collected data. OBJECTIVES: Explore the floor and ceiling effects, convergent, and divergent validity of the International Spinal Cord Injury Basic Quality of Life Data Set (SCI QoL-BDS) in a sample of people with spinal cord damage (SCD) from different countries, with different causes (both traumatic and non-traumatic), and different settings. SETTING: Community dwellers with SCD in Australia, Brazil, India, The Netherlands, and USA, and inpatient rehabilitation: India. METHODS: Adults (>18 years) with chronic SCD with either traumatic or non-traumatic aetiologies living in the community (n = 624), in inpatient rehabilitation following the onset of SCI (India; n = 115) and able-bodied controls (Australia; n = 220) had the following data collected by survey or face-face interview: SCI QoL-BDS, demographic and clinical characteristics (e.g., age, gender, years post SCI/SCD, education, employment) and reference measures of quality of life, disability and depression. RESULTS: For the whole sample, there were no notable floor or ceiling effects, internal consistency was good (Cronbach's alpha = 0.84) and the corrected item-total correlations generally were acceptable (all > 0.3 except for in Brazilian cohort). Convergent and divergent validity were largely confirmed though there were some aspects of validity that were suboptimal. CONCLUSIONS: Only minor psychometric issues were identified. This preliminary analysis suggests that there are no reason for concern about the use of the SCI QoL-BDS for clinical or research purposes, notwithstanding the need for further studies.

Development and Feasibility of SymTrak, aMulti-domain Tool for Monitoring Symptoms of Older Adults in Primary Care.

BACKGROUND: A clinically practical, brief, user-friendly, multi-domain self-report and caregiver-report tool is needed for tracking actionable symptoms in primary care for elderly patients with multiple chronic conditions (MCCs). OBJECTIVE: Develop and assess usability, administration time, and internal reliability of SymTrak. DESIGN AND PARTICIPANTS: Phase I: legacy instruments, content validity, analyses of existing data, focus groups (physicians, nurses, patients, informal caregivers), and Think Aloud interviews (patients, caregivers) were used to develop SymTrak. Phase II (pilot feasibility study): 81 (27 patient-caregiver dyads, 27 patients without an identified caregiver) participants were self-administered SymTrak in clinic. MAIN MEASURES: SymTrak and demographic questions. KEY RESULTS: Consistent themes emerged from phase I focus groups. Ambiguous wording was corrected with Think Aloud feedback. In phase II, patients and caregivers preferred circling words instead of numbers for item response options. SymTrak self-administration completion time in clinic was brief; mean was 2.4, 3.0, and 3.3 min for the finalized circlingwords version, respectively, for caregivers, dyadic patients, and patients without a caregiver; and the maximum was 6.2 min for any participant. Usability questionnaire ratings were high. Cronbach's alpha for the SymTrak 23-item total score was 0.86, 0.79, and 0.81 for caregivers, dyadic patients, and patients without a caregiver, respectively. CONCLUSIONS: SymTrak demonstrates content validity, positive qualitative findings, high perceived usability, brief self-administered completion time, and good internal reliability.

A critique of validity analysis from instruments assessing the affective domain.

BACKGROUND: Schools and colleges of pharmacy need to show evidence that their students have internalized professional values, and many choose to do so through quantitative instruments. A review of the literature was completed to identify the evidence of validity of the scores from instruments designed to assess pharmacy students in the affective domain. METHODS: Electronic databases were searched to identify instruments. Basic information regarding the instruments, the facets of validity assessed, and the evidence for validity were reviewed. RESULTS: Of the studies identified, 25 focused on assessing the affective domain and reported evidence of at least one facet of validity. Most reported evidence of validity from two or more sources, and most reported evidence concerning test content and internal structure (i.e. internal consistency reliability or factor analysis). Other sources of validity were missing from most studies. IMPLICATIONS: More research is needed to investigate the validity of the scores of instruments developed to assess pharmacy students within the affective domain, especially regarding relations to other variables, response processes, and consequences of use.

Modeling the intersectionality of processes in the social production of health inequalities.

BACKGROUND: The recent pair of studies by Bauer and Scheim make substantial contributions to the literature on intersectionality and health: a validation study of the Intersectional Discrimination Index and a study outlining a promising analytic approach to intersectionality that explicitly considers the roles of social processes in the production of health inequalities. RATIONALE: In this commentary, I situate Bauer and Scheim's contribution within the wider landscape of intersectional scholarship. I also respond to emerging concerns about the value of descriptive intersectional approaches, in particular the critique that such approaches blunt the critical edge and transformative aims of intersectionality. Finally, I outline important future directions for intersectional scholarship modeling social processes, in particular, the need for addressing structural determinants of inequalities intersectionally. CONCLUSIONS: Whether a study is descriptive or analytic, engagement with theory is essential in order to maintain the critical and transformative edge of intersectionality. Theories of population health such as fundamental causes, social production, and ecosocial theory, should be framed and applied in explicitly intersectional terms. As the field moves toward intersectional evaluations of social processes, attention should be given to all ecological levels but especially the structural/institutional level. This attention includes considering interactions between intersectional social strata and contexts and considering the roles of structural-level discrimination in shaping population health outcomes intersectionally.

Neuropsychological tests of the future: How do we get there from here?

OBJECTIVE: This article reviews current approaches to neuropsychological assessment, identifies opportunities for development of new methods using modern psychometric theory and advances in technology, and suggests a transition path that promotes application of novel methods without sacrificing validity. METHODS: Theoretical/state-of-the-art review. CONCLUSIONS: Clinical neuropsychological assessment today does not reflect advances in neuroscience, modern psychometrics, or technology. Major opportunities for improving practice include both psychometric and technological strategies. Modern psychometric approaches including item response theory (IRT) enable linking procedures that can place different measures on common scales; adaptive testing algorithms that can dramatically increase efficiency of assessment; examination of differential item functioning (DIF) to detect measures that behave differently in different groups; and person fit statistics to detect aberrant patterns of responding of high value for performance validity testing. Opportunities to introduce novel technologies include computerized adaptive testing, Web-based assessment, healthcare- and bio-informatics strategies, mobile platforms, wearables, and the 'internet-of-things'. To overcome inertia in current practices, new methods must satisfy requirements for back-compatibility with legacy instrumentation, enabling us to leverage the wealth of validity data already accrued for classic procedures. A path to achieve these goals involves creation of a global network to aggregate item-level data into a shared repository that will enable modern psychometric analyses to refine existing methods, and serve as a platform to evolve novel assessment strategies, which over time can revolutionize neuropsychological assessment practices world-wide.